haskell: pkgsStatic fixes #162374

rnhmjoj · 2022-03-01T14:36:24Z

Motivation for this change

Fixing common build failures in pkgsStatic.haskellPackages

See the commit messages for details.

Things done

built static GHCs 9.0.2 and 9.2.1
built pkgsStatic.haskellPackages.{QuickCheck,pretty-simple} (previously failing)
built and tested pkgsStatic.haskellPackages.pandoc

pkgs/development/haskell-modules/generic-builder.nix

sternenseemann · 2022-03-01T16:39:52Z

I'd drop d8dcb588aed526a632c3060c1b091908c7cf31df, since it changes the meaning of haskell.packages.ghc* / haskellPackages between pkgs and pkgsStatic which I find problematic. Especially since the compiler build itself is not affected, this doesn't seem necessary?
Can you add a package to the staticHaskellPackages jobset in pkgs/top-level/release-haskell.nix which doesn't build without -fexternal-dynamic-refs? This would help keeping tabs on it.

rnhmjoj · 2022-03-02T00:10:54Z

Especially since the compiler build itself is not affected, this doesn't seem necessary?

I see your point, though I'm not sure how useful the default compiler is, given most packages fails to build. Ideally, we should fix the linking errors, but avoiding gmp is the only way I've found so far.

Can you add a package to the staticHaskellPackages jobset in pkgs/top-level/release-haskell.nix which doesn't build without -fexternal-dynamic-refs? This would help keeping tabs on it.

Done.

cdepillabout · 2022-03-02T02:23:34Z

pkgs/development/compilers/ghc/9.2.1.nix

+    GhcLibHcOpts += -fPIC -fexternal-dynamic-refs
+    GhcRtsHcOpts += -fPIC -fexternal-dynamic-refs


Would you be able to add a comment here why -fexternal-dynamic-refs is needed for enableRelocatedStaticLibs?

From the GHC manual: https://downloads.haskell.org/ghc/latest/docs/html/users_guide/phases.html#ghc-flag--fexternal-dynamic-refs

When generating code, assume that entities imported from a different module might be dynamically linked. This flag is enabled automatically by -dynamic.

It is not clear to me why this would be necessary if we are statically linking things. (Although I don't have a good understanding of static linking, so maybe it should be obvious.)

I have to admit I'm baffled by this too and was kind of expecting the haskell maintainers would know better.
Basically, I'm implementing the solution proposed in this blog post: https://www.tweag.io/blog/2020-09-30-bazel-static-haskell/.

Thanks for the link to the blog post. It says:

When evaluating template Haskell splices, GHC will execute compiled expressions in its built-in bytecode interpreter and this code has to be compatible with the RTS of GHC itself. In short, a GHC configured with a dynamic RTS will not be able to load static Haskell libraries to evaluate template Haskell splices.

It then goes on to say:

We need to build GHC with a static RTS and to make sure that Haskell code is compiled as position independent code so that it can be loaded into a running GHC for template Haskell splices.

Both of these statements make sense of course, but they don't seem to explain why -fexternal-dynamic-refs is necessary.

The GHC man page reads:

Generate code for linking against dynamic libraries

I assume this means that the RTS will have some kind of dynamic library loading facilities even of static (Haskell?) archives, so the byte code interpreter / whatever used by TH will actually work. -fPIC also serves this purpose.

cdepillabout · 2022-03-02T02:28:21Z

Maybe I'm misinterpreting, but it seems that this change allows building Haskell packages from pkgsStatic.haskellPackages.* that use Template Haskell? This seems like quite a large improvement, and would potentially be game-changing for a lot of projects.

rnhmjoj · 2022-03-02T09:27:22Z

allows building Haskell packages from pkgsStatic.haskellPackages.* that use Template Haskell?

Yes, correct.

pkgs/development/compilers/ghc/9.0.2.nix

cdepillabout

Aside from the above comment suggestions, this LGTM!

I tried nix-build -A pkgsStatic.pretty-simple and it appears to build and run correctly!

cdepillabout · 2022-03-03T11:29:17Z

@sternenseemann Does this look alright to merge to you?

sternenseemann · 2022-03-03T11:52:45Z

I see your point, though I'm not sure how useful the default compiler is, given most packages fails to build. Ideally, we should fix the linking errors, but avoiding gmp is the only way I've found so far.

Have you tried actually building anything that uses FFI in TH? I guess that would need a dynamically linked library as well?

I'm still opposed to changing the meaning of haskell.packages.ghc*, but I would be fine with changing the meaning of top level haskellPackages as a compromise, i.e.:

  haskellPackages = dontRecurseIntoAttrs (
    if stdenv.hostPlatform.isStatic
    then haskell.packages.native-bignum.ghc902
    else haskell.packages.ghc902
  );

  /* … */
  ghc = targetPackages.haskellPackages.ghc or (
    if stdenv.targetPlatform.isStatic
    then haskell.compiler.native-bignum.ghc902
    else haskell.compiler.ghc902
  );

@sternenseemann Does this look alright to merge to you?

~~FWIW, eval is failing.~~ Edit: unrelated.

rnhmjoj · 2022-03-03T14:54:39Z

but I would be fine with changing the meaning of top level haskellPackages as a compromise,

Looks great to me! Thanks for the suggestion.

pkgs/development/haskell-modules/generic-builder.nix

pkgs/top-level/all-packages.nix

cdepillabout · 2022-03-04T01:41:51Z

I would be fine with changing the meaning of top level haskellPackages as a compromise

I agree this was a good suggestion. I think the current state of this PR makes a lot of sense, having pkgsStatic.haskellPackages explicitly pointing at pkgsStatic.haskell.packages.native-bignum.ghc-902.

Have you tried actually building anything that uses FFI in TH? I guess that would need a dynamically linked library as well?

That's a good question. Do you happen to know a package that uses FFI in TH off the top of your head? I gave it some thought, but I couldn't come up with anything. (I did try compiling pkgsStatic.haskellPackages.zlib, which does successfully compile, but I imagine it is unlikely to use TH.)

Although even if this doesn't work, I don't feel like it should hold up merging this PR, since this PR currently improves the status quo.

rnhmjoj · 2022-03-08T10:49:58Z

So, can we merge this?

sternenseemann · 2022-03-08T11:58:21Z

Probably, I'll try to have a final look soon. A small request: Could you use haskellPackages.mkDerivation over haskell-generic-builder in the commit message? I'd like to have correspondence to an attribute where possible. Rebasing on haskell-updates would be useful anyways, so CI wouldn't be broken by an unrelated problem.

rnhmjoj · 2022-03-08T14:26:47Z

Commit message updated and rebased.

pkgs/top-level/all-packages.nix

This change allows loading statically compiled libreries into a running GHC, thus fixing the build of haskell packages that use TemplateHaskell. See [1] for the details. Fixes issue NixOS#61575, NixOS#124284. [1]: https://www.tweag.io/blog/2020-09-30-bazel-static-haskell/

pkgs/development/haskell-modules/generic-builder.nix

This change makes the haskell builder run the haddockPhase only if the haddock program is availble (for example, it's not when cross-compiling).

This works around several build failures where programs attempt to link to the libgmp.so (which is not available), such as: Error loading shared library libgmp.so: No such file or directory

rnhmjoj · 2022-03-17T18:54:30Z

@sternenseemann Thank you for reviewing!

nixos-discourse · 2022-03-23T00:52:53Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/statically-linking-haskell-fails-to-link-libffi/18317/2

nixos-discourse · 2022-03-24T10:42:25Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/statically-linking-haskell-fails-to-link-libffi/18317/3

rnhmjoj requested review from cdepillabout, sternenseemann, maralorn and expipiplus1 as code owners March 1, 2022 14:36

rnhmjoj changed the title ~~haskell:~~ haskell: pkgsStatic fixes Mar 1, 2022

github-actions bot added the 6.topic: haskell label Mar 1, 2022

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Mar 1, 2022

sternenseemann reviewed Mar 1, 2022

View reviewed changes

pkgs/development/haskell-modules/generic-builder.nix Outdated Show resolved Hide resolved

rnhmjoj force-pushed the haskell-updates branch from d8dcb58 to cf15c9a Compare March 2, 2022 00:11

cdepillabout reviewed Mar 2, 2022

View reviewed changes

cdepillabout reviewed Mar 3, 2022

View reviewed changes

pkgs/development/compilers/ghc/9.0.2.nix Outdated Show resolved Hide resolved

cdepillabout approved these changes Mar 3, 2022

View reviewed changes

rnhmjoj force-pushed the haskell-updates branch from cf15c9a to 27e2010 Compare March 3, 2022 10:06

rnhmjoj force-pushed the haskell-updates branch from 27e2010 to 7dc3171 Compare March 3, 2022 14:54

sternenseemann requested changes Mar 3, 2022

View reviewed changes

pkgs/development/haskell-modules/generic-builder.nix Outdated Show resolved Hide resolved

pkgs/top-level/all-packages.nix Outdated Show resolved Hide resolved

rnhmjoj force-pushed the haskell-updates branch from 7dc3171 to f1ce00f Compare March 3, 2022 21:04

rnhmjoj force-pushed the haskell-updates branch from f1ce00f to 131616d Compare March 8, 2022 14:24

sternenseemann requested changes Mar 9, 2022

View reviewed changes

pkgs/top-level/all-packages.nix Show resolved Hide resolved

rnhmjoj force-pushed the haskell-updates branch from 131616d to 7244522 Compare March 9, 2022 20:08

rnhmjoj force-pushed the haskell-updates branch from 7244522 to adb4686 Compare March 9, 2022 22:06

sternenseemann requested changes Mar 12, 2022

View reviewed changes

pkgs/development/haskell-modules/generic-builder.nix Outdated Show resolved Hide resolved

rnhmjoj force-pushed the haskell-updates branch from adb4686 to 708ae5b Compare March 15, 2022 11:03

rnhmjoj added 3 commits March 17, 2022 17:18

haskellPackages.mkDerivation: check haddock availability

87e418c

This change makes the haskell builder run the haddockPhase only if the haddock program is availble (for example, it's not when cross-compiling).

release-haskell.nix: test package that needs -fexternal-dynamic-refs

9afd54a

pkgsStatic.{haskellPackages,ghc}: default to BigNum

a503a02

This works around several build failures where programs attempt to link to the libgmp.so (which is not available), such as: Error loading shared library libgmp.so: No such file or directory

rnhmjoj force-pushed the haskell-updates branch from 708ae5b to a503a02 Compare March 17, 2022 16:18

sternenseemann merged commit 299bfb1 into NixOS:haskell-updates Mar 17, 2022

sternenseemann mentioned this pull request Apr 7, 2022

Fully static Haskell executables - overview issue #43795

Open

60 tasks

This was referenced Apr 9, 2022

haskell-updates: pretty-simple static cross build broken #124284

Closed

pkgsStatic build fail: [genericBuild: command not found] #61486

Closed

cdepillabout mentioned this pull request Apr 23, 2022

Fix enableRelocatedStaticLibs. input-output-hk/haskell.nix#1430

Closed

cdepillabout mentioned this pull request May 6, 2022

example GitHub workflow for for building Spago statically purescript/spago#882

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

haskell: pkgsStatic fixes #162374

haskell: pkgsStatic fixes #162374

rnhmjoj commented Mar 1, 2022 •

edited

Loading

sternenseemann commented Mar 1, 2022

rnhmjoj commented Mar 2, 2022

cdepillabout Mar 2, 2022

rnhmjoj Mar 2, 2022

cdepillabout Mar 3, 2022

sternenseemann Mar 3, 2022

cdepillabout commented Mar 2, 2022

rnhmjoj commented Mar 2, 2022

cdepillabout left a comment

cdepillabout commented Mar 3, 2022

sternenseemann commented Mar 3, 2022 •

edited

Loading

rnhmjoj commented Mar 3, 2022

cdepillabout commented Mar 4, 2022

rnhmjoj commented Mar 8, 2022

sternenseemann commented Mar 8, 2022

rnhmjoj commented Mar 8, 2022

rnhmjoj commented Mar 17, 2022

nixos-discourse commented Mar 23, 2022

nixos-discourse commented Mar 24, 2022

		GhcLibHcOpts += -fPIC -fexternal-dynamic-refs
		GhcRtsHcOpts += -fPIC -fexternal-dynamic-refs

haskell: pkgsStatic fixes #162374

haskell: pkgsStatic fixes #162374

Conversation

rnhmjoj commented Mar 1, 2022 • edited Loading

Motivation for this change

Things done

sternenseemann commented Mar 1, 2022

rnhmjoj commented Mar 2, 2022

cdepillabout Mar 2, 2022

Choose a reason for hiding this comment

rnhmjoj Mar 2, 2022

Choose a reason for hiding this comment

cdepillabout Mar 3, 2022

Choose a reason for hiding this comment

sternenseemann Mar 3, 2022

Choose a reason for hiding this comment

cdepillabout commented Mar 2, 2022

rnhmjoj commented Mar 2, 2022

cdepillabout left a comment

Choose a reason for hiding this comment

cdepillabout commented Mar 3, 2022

sternenseemann commented Mar 3, 2022 • edited Loading

rnhmjoj commented Mar 3, 2022

cdepillabout commented Mar 4, 2022

rnhmjoj commented Mar 8, 2022

sternenseemann commented Mar 8, 2022

rnhmjoj commented Mar 8, 2022

rnhmjoj commented Mar 17, 2022

nixos-discourse commented Mar 23, 2022

nixos-discourse commented Mar 24, 2022

rnhmjoj commented Mar 1, 2022 •

edited

Loading

sternenseemann commented Mar 3, 2022 •

edited

Loading